Image processing device and method, image processing system, and image processing program

ABSTRACT

An image processing device includes: an entire image display control portion that performs control to display an entire image of a predetermined region in an entire image display window; and a cutout image display control portion that performs control to enlarge a plurality of tracking subjects included in the entire image and display the tracking subjects in a cutout image display window. The cutout image display control portion performs the control in such a manner that one cutout image including the tracking subjects is displayed in the cutout image display window in a case where relative distances among the tracking subjects are equal to or smaller than a predetermined value, and that two cutout images including the respective tracking subjects are displayed in the cutout image display window in a case where the relative distances among the tracking subjects are larger than the predetermined value.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing device and method, animage processing system, and an image processing program for trackingand displaying a plurality of moving objects.

2. Description of Related Art

There is an image processing system that displays a moving object bytracking the moving object using a monitoring camera or the like.JP-A-2006-121261 describes a technique of calculating pan-tilt angleinformation of a swivel base on which is mounted a camera according toposition information and height information of a moving object obtainedfrom a GPS (Global Positioning System) and controlling the camera withthe angle of the swivel base so as not to lose track of the movingobject using the single camera. Also, a technique of tracking aplurality of moving objects is described in “Kakushindo-tsukikasetsugunno sougosayounimotozuku fukusuutaishou tsuiseki” ComputerVision and Image Media vol. 43 No. SIG4 (CVIM 4), June 2006.

SUMMARY OF THE INVENTION

When a plurality of moving objects are tracked by a single camera, theremay be a case where a moving object is no longer tracked depending onmotions of the moving object. In order to eliminate this inconvenience,each of a plurality of the moving objects may be tracked by a differentcamera. In this case, however, it is quite difficult to keep track ofthe positional relation among all the moving objects within imagesgenerated by the respective cameras.

As a technique of tracking detected moving objects while keeping trackof the positional relation, a radar system is generally used. The radarsystem, however, provides poor visual information and is therefore notsuitable to identify a moving object.

There is another technique of tracking and displaying a plurality ofmoving objects by extracting the moving objects individually in thetracking processing. According to such a technique, in a case where twomoving objects overlap, the overlapping moving objects are tracked asone moving object and when the two moving objects are separated, the twomoving objects are recognized as newly generated moving objects. It istherefore difficult to track a plurality of moving objects precisely.

In a case where a plurality of moving objects are tracked and all themoving objects are displayed on a single screen, the display size ofeach moving object on the screen becomes too small to identify eachmoving object when the moving objects are distant very much from oneanother. Meanwhile, when a window is provided to each moving object todisplay each moving object, in a case where a plurality of the movingobjects come close together, the same image is displayed in more thanone window. It thus becomes difficult to use the display screeneffectively.

It is therefore desirable to use the display screen effectively whilemaking it easier for the user to recognize respective moving objects inthe tracking and displaying processing for a plurality of movingobjects.

According to an embodiment of the present invention, there is providedan image processing device including an entire image display controlportion that performs control to display an entire image of apredetermined region in an entire image display window, and a cutoutimage display control portion that performs control to enlarge aplurality of tracking subjects included in the entire image and displaythe tracking subjects in a cutout image display window. The cutout imagedisplay control portion performs the control in such a manner that onecutout image including the plurality of tracking subjects is displayedin the cutout image display window in a case where relative distancesamong the plurality of tracking subjects are equal to or smaller than apredetermined value and performs the control in such a manner that twocutout images including the respective tracking subjects are displayedin the cutout image display window in a case where the relativedistances among the plurality of tracking subjects are larger than thepredetermined value.

According to another embodiment of the present invention, there isprovided an image processing system including an image transmissiondevice having a camera portion that generates image data, an image dataprocessing portion that generates compressed image data by encoding theimage data generated by the camera portion, and a data transmissionportion that transmits the compressed image data generated by the imagedata processing portion, and an image processing device having a datareception portion that receives the compressed image data, an entireimage display control portion that performs control to display an entireimage of a predetermined range generated from image data obtained bydecoding the compressed image data received at the data receptionportion in an entire image display window, and a cutout image displaycontrol portion that performs control to enlarge a plurality of trackingsubjects included in the entire image and display the tracking subjectsin a cutout image display window. The cutout image display controlportion performs the control in such a manner, that one cutout imageincluding the plurality of tracking subjects is displayed in the cutoutimage display window in a case where relative distances among theplurality of tracking subjects are equal to or smaller than apredetermined value and performs the control in such a manner that twocutout images including the respective tracking subjects are displayedin the cutout image display window in a case where the relativedistances among the plurality of tracking subjects are larger than thepredetermined value.

According to still another embodiment of the present invention, there isprovided an image processing method including the steps of performingcontrol to display an entire image of a predetermined region in anentire image display window and performing control to enlarge aplurality of tracking subjects included in the entire image and displaythe tracking subjects in a cutout image display window. In the step ofdisplaying the tracking subjects, the control is performed in such amanner that one cutout image including the plurality of trackingsubjects is displayed in the cutout image display window in a case whererelative distances among the plurality of tracking subjects are equal toor smaller than a predetermined value and the control is performed insuch a manner that two cutout images including the respective trackingsubjects are displayed in the cutout image display window in a casewhere the relative distances among the plurality of tracking subjectsare larger than the predetermined value.

According to still another embodiment of the present invention, there isprovided an image processing program causing a computer to perform thesteps of performing control to display an entire image of apredetermined region in an entire image display window and performingcontrol to enlarge a plurality of tracking subjects included in theentire image and display the tracking subjects in a cutout image displaywindow. In the step of displaying the tracking subjects, the control isperformed in such a manner that one cutout image including the pluralityof tracking subjects is displayed in the cutout image display window ina case where relative distances among the plurality of tracking subjectsare equal to or smaller than a predetermined value and the control isperformed in such a manner that two cutout images including therespective tracking subjects are displayed in the cutout image displaywindow in a case where the relative distances among the plurality oftracking subjects are larger than the predetermined value.

According to the embodiments of the present invention, the user becomesable to obtain detailed images of the tracking subjects that have notbeen obtained by the wide angle imaging while keeping track of thepositional relation among the tracking subjects in the entire imagedisplay window. According to the embodiments of the present invention,in a case where the relative distances between the respective trackingsubjects are equal to or smaller than the predetermined threshold value,the user becomes able to track a plurality of the tracking subjects as agroup by displaying all the tracking subjects in one cutout imagedisplay window. According to the embodiments of the invention, even in acase where the relative distances among a plurality of the respectivetracking subjects are larger than the predetermined value, by displayingtwo cutout images including the respective tracking subjects in thecutout image display window, the user becomes able to keep track of adetailed image for each of a plurality of the tracking subjects.

As has been described, according to the embodiments of the presentinvention, even when there are a large number of tracking subjects, theuser becomes able to keep track of the tracking subjects preciselyaccording to the positional relation among the tracking subjectsdisplayed in the entire image display window.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing an image processing system according to anembodiment to which the present invention is applied;

FIG. 2 is a view showing the internal configuration of an imagetransmission device and an image processing device;

FIG. 3 is a view showing an example of a display screen shown on adisplay provided to the image processing device;

FIG. 4 is a flowchart used to describe a moving object tracking anddisplaying processing operation on the basis of moving object detection;

FIG. 5 is a view showing an example of a display screen shown on adisplay provided to the image processing device;

FIG. 6 is a view showing a manner in which a plurality of moving objectsare cut out and displayed on a display screen shown on a displayprovided to the image processing device;

FIG. 7 is a view showing a manner in which a plurality of moving objectsare cut out and displayed on a display screen shown on a displayprovided to the image processing device;

FIG. 8 is a view showing a manner in which a plurality of moving objectsare cut out and displayed on a display screen shown on a displayprovided to the image processing device;

FIG. 9 is a view showing a manner in which a plurality of moving objectsare cut out and displayed on a display screen shown on a displayprovided to the image processing device;

FIG. 10 is a view showing a display screen at SXGA of 1280×1024 dots;

FIG. 11 is a view showing an example of a display screen on whichtracking subjects imaged by an HD camera are displayed;

FIG. 12 is a view showing an example of a display screen on whichtracking subjects imaged by an HD camera are displayed;

FIG. 13 is a view showing an example of a display screen on whichtracking subjects imaged by an HD camera are displayed;

FIG. 14 is a flowchart used to describe a moving object tracking anddisplaying processing operation according to latitude and longitudeinformation;

FIG. 15 is a view showing the positional relation among the imagetransmission device, a reference subject, and a moving object;

FIG. 16 is a view showing an entire image display window displayed bythe image transmission device;

FIG. 17 is a view showing a manner in which a predetermined number ofcontinuous images among a plurality of generated images are displayed inthe entire image display window and an image of a predetermined cutoutrange in the image being displayed in the entire image display window isdisplayed in a cutout image display window;

FIG. 18 is a view showing the configuration of an image processingsystem according to another embodiment to which the present invention isapplied;

FIG. 19 is a view showing the internal configuration of an imagetransmission device and an image processing device;

FIG. 20 is a view showing a panoramic entire image; FIG. 21 is a viewshowing the internal configuration of the image transmission deviceprovided to the image processing system;

FIG. 22 is a view showing an example of a display screen shown on adisplay provided to the image processing device;

FIG. 23 is a view showing an example of a display screen shown on adisplay provided to the image processing device; and

FIG. 24 is a view showing an example of a display screen shown on adisplay provided to the image processing device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, concrete embodiments to which the present invention isapplied will be described in detail with reference to the drawings.

FIG. 1 is a view showing an image processing system 1 according to anembodiment to which the present invention is applied. The imageprocessing system 1 includes an image transmission device 2 and an imageprocessing device 3 connected to each other via a network 4, such as aLAN (Local Area Network) and the Internet.

FIG. 2 is a view showing the internal configuration of the imagetransmission device 2 and the image processing device 3. The imagetransmission device 2 transmits an image signal generated from animaging signal to the image processing device 3 via the network 4. Theimage processing device 3 applies predetermined signal processing to theimage signal received from the image transmission device 2 via thenetwork 4. Then, the image processing device 3 makes an entire image,which is an image of the imaging region, and a cutout image, which is animage obtained by cutting out a region including moving objects astracking subjects from the entire image, be displayed in differentwindows.

The image transmission device 2 includes a camera portion 10, an imagedata processing portion 11, an external sensor 12, and a datatransmission portion 13. The camera portion 10 includes an imagingportion 101 and an imaging signal processing portion 102. The image dataprocessing portion 11 includes an encoder (ENC) 103, a moving object andstationary object detection portion 104, and a metadata generationportion 105.

In the camera portion 10, the imaging portion 101 has an imaging lens(not shown) and an imaging element (not shown) formed of a CCD (ChargedCoupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). Inthe imaging portion 101, the imaging element captures an image of apredetermined region via the imaging lens to generate an imaging signaland supplies the imaging signal processing portion 102 with the imagingsignal. The imaging signal processing portion 102 applies processing,such as sample hold, gain control, and A/D (Analog-to-Digital)conversion, and processing, such as a white balance adjustment and agamma correction, to the imaging signal supplied from the imagingportion 101. The imaging signal processing portion 102 supplies theencoder 103 and the moving object and stationary object detectionportion 104 in the image data processing portion 11 with image datagenerated through the processing specified above.

The encoder 103 applies encoding processing by the MPEG (Moving PictureExperts Group)-4 coding method to the image data supplied from theimaging signal processing portion 102 and supplies the data transmissionportion 13 with compressed image data.

The moving object and stationary object detection portion 104 performsprocessing to detect a moving object and a stationary object from theimage data supplied from the imaging signal processing portion 102. Themoving object and stationary object detection portion 104 supplies themetadata generation portion 105 with detection information of the movingobject and the stationary object.

Detection information of a moving object from the external sensor 12 isalso supplied to the metadata generation portion 105. The externalsensor 12 is formed, for example, of an infrared sensor, a temperaturesensor, or a sound sensor. The external sensor 12 supplies the metadatageneration portion 105 with information informing the detection of amoving object on the basis of an infrared ray, a temperature, or asound. The metadata generation portion 105 generates metadata includingdetection information of the moving object and the stationary objectsupplied from the moving object and stationary object detection portion104 and the detection information from the external sensor 12 andsupplies the data transmission portion 13 with the metadata.

The data transmission portion 13 transmits the image data supplied fromthe encoder 103 to the image processing device 3 via the network 4according to the TCP/IP (Transmission Control Protocol/InternetProtocol). The data transmission portion 13 also transmits the metadatasupplied from the metadata generation portion 105 to the imageprocessing device 3 via the network 4.

The image processing device 3 includes a data reception portion 111, aposition information extraction portion 112, an entire image generationportion 113, a cutout image generation portion 114, an entire imagedisplay portion 115, and a cutout image display portion 116.

Although it is not shown in the drawing, the image processing device 3includes a CPU (Central Processing Unit), a working RAM (Random AccessMemory), a ROM (Read Only Memory) that stores an image processingprogram according to one embodiment of the present invention, data, andso forth. By running the image processing program on the CPU, a seriesof processing by the image processing device 3 is performed. The imageprocessing program may be recorded, for example, in a recording medium,such as a CD-ROM, so that it can be provided from the outside.

The data reception portion 111 receives the image data and the metadatafrom the image transmission device 2. The data transmission portion 111supplies the entire image generation portion 113 with the received imagedata. Also, the data reception portion 111 supplies the positioninformation extraction portion 112 with the received image data andmetadata.

The position information extraction portion 112 specifies an image framein which moving objects are detected from the image data received at thedata reception portion 111 according to the metadata of the detectioninformation of the moving object and the stationary object. The imageprocessing device 3 further includes an operation portion (not shown)used to specify a plurality of tracking subjects in response to anoperation by the user from the entire image being displayed in theentire image display window. Of the moving objects, moving objectsdefined as the tracking subjects are set when specified in response toan operation by the user. A setting signal informing the setting issupplied to the position information extraction portion 112.

The position information extraction portion 112 applies predeterminedarithmetic processing to the image frame in which the moving objects aredetected to extract the position information of the moving objects asthe tracking subjects in the image and supplies the cutout imagegeneration portion 114 with the extracted position information. Whichmoving objects are set as the tracking subjects is specified, forexample, by a setting operation by the user. Also, the positioninformation extraction portion 112 supplies the cutout image generationportion 114 with the image data received at the data reception portion111.

Alternatively, it may be configured in such a manner that the positioninformation extraction portion 112 takes in the position information ofthe tracking subjects from the outside using the GPS (Global PositioningSystem) and performs processing to convert the position information ofthe tracking subjects to the position information in the image.

The entire image generation portion 113 supplies the entire imagedisplay portion 115 with an image of the image data supplied from thedata reception portion 111 as the entire image.

The cutout image generation portion 114 calculates relative distancesbetween the respective tracking subjects according to the positioninformation of the tracking subjects supplied from the positioninformation extraction portion 112 and determines whether the relativedistances are larger than a predetermined value, that is, apredetermined threshold value. The relative distances are calculatedaccording to either one or both of the sizes of the tracking subjectswithin the entire image and distances between the respective trackingsubjects within the entire image. The cutout image generation portion114 then specifies a cutout range on the basis of this determination andsupplies the entire image generation portion 113 with the information ofthis cutout range as angle-of-view information. The entire imagegeneration portion 113 cuts out an image of the cutout range accordingto the angle-of-view information supplied from the cutout imagegeneration portion 114 from the entire image and it supplies the cutoutimage generation portion 114 with the cutout image.

In a case where the relative distances between the respective trackingsubjects calculated according to the position information extracted bythe position information extraction portion 112 are equal to or smallerthan the predetermined threshold value, the cutout image generationportion 114 specifies one cutout range including all the trackingsubjects having the relative distances between the respective trackingsubjects equal to or smaller than the predetermined threshold valueaccording to the position information. The cutout image generationportion 114 then generates an image of this cutout range as a cutoutimage. Meanwhile, in a case where the relative distances between therespective tracking subjects are larger than the predetermined thresholdvalue, the cutout image generation portion 114 specifies two cutoutranges including the respective tracking subjects having the relativedistances far larger than the predetermined threshold value according tothe position information. The cutout image generation portion 114 thengenerates images of the specified cutout ranges as cutout images.

The cutout image generation portion 114 enlarges the generated cutoutimage(s) at an arbitrary enlargement ratio in response to an operationby the user or according to its own determination. Also, the cutoutimage generation portion 114 may reduce the image enlarged once at anarbitrary reduction ratio in response to an operation by the user oraccording to its own determination. The cutout image generation portion114 supplies the cutout image display portion 116 with the generatedcutout image(s). The cutout image generation portion 114 is capable ofchanging the enlargement ratio or the reduction ratio of the cutoutimage(s) arbitrarily in response to a setting operation by the user. Inaddition, the cutout range(s) is specified according to the positioninformation of the tracking subjects extracted by the positioninformation extraction portion 112 so as to include the trackingsubjects completely.

It should be noted that the cutout image generation portion 114 iscapable of generating three or more cutout images by repetitivelyperforming the arithmetic processing between the respective trackingsubjects according to the position information of the tracking subjects.

The image processing device 3 includes a display, such as an LCD (LiquidCrystal Display). The display provided to the image processing device 3displays, for example, a display screen 21 shown in FIG. 3. On thedisplay screen 21, the entire image of a predetermined region isdisplayed in an entire image display window 22. Also, on the displayscreen 21, a cutout image(s) cut out from the entire image beingdisplayed in the entire image display window 22 is displayed in a cutoutimage display window 24. The entire image display portion 115 displaysthe entire image supplied from the entire image generation portion 113in the entire image display window 22. The cutout image display portion116 displays the cutout image(s) supplied from the cutout imagegeneration portion 114 in the cutout image display window 24.

In this manner, the image processing device 3 includes the entire imagegeneration portion 113 and the entire image display portion 115 andperforms control to display the entire image of the predetermined regionin the entire image display window. Also, the image processing device 3includes the cutout image generation portion 114 and the cutout imagedisplay portion 116 and performs control to enlarge a plurality oftracking subjects included in the entire image and display the trackingsubjects in the cutout image display window.

It should be appreciated that an image of the image data transmittedfrom the image transmission device 2 can be either a moving image or astill image.

In a case where the image is a moving image, an image can be displayedin real time in the entire image display window and the cutout imagedisplay window.

In a case where the image is a still image, a cutout image displayed inthe cutout image display window is an image coinciding with the entireimage being displayed in the entire image display window. Alternatively,it may be configured in such a manner that a cutout image displayed inthe cutout image display window is updated at arbitrarily set timeintervals. In this case, the cutout image display portion 116 updates animage to be displayed in the cutout image display window in response toa setting operation by the user or at every arbitrary time according toits own determination.

An operation example of moving object tracking and displaying processingby the image processing system 1 will now be described. FIG. 4 is aflowchart used to describe the moving object tracking and displayingprocessing operation on the basis of moving object detection.

In Step S1, the position information extracting portion 112 starts theposition information extraction processing by the moving objectdetection processing.

In Step S2, from an image frame supplied from the data reception portion111, the position information extraction portion 112 extracts positioninformation of moving objects as tracking subjects in the image frameincluding moving objects according to the metadata of the moving objectdetection information supplied from the metadata generation portion 105.

For example, as is shown in FIG. 5, the moving object and stationaryobject detection portion 104 detects moving objects 31 through 33 in animaging region corresponding to the display screen 21. In this case, themetadata generation portion 105 generates metadata of informationinforming the detection of the moving objects 31 through 33. Theposition information extraction portion 112 specifies an image frameincluding the moving objects 31 through 33 using the image data suppliedfrom the data reception portion 111. Herein, assume that the movingobjects 31 through 33 are all set as tracking subjects in response to asetting operation by the user. A setting signal informing the setting issupplied to the position information extraction portion 112.

In Step S3, the position information extraction portion 112 calculates acoordinate (upper left end point) corresponding to the upper left end ofan image region encompassing a tracking subject and a coordinate (lowerright end point) corresponding to the lower right end in the specifiedimage frame. Herein the coordinate is expressed by (x, y).

In Step S4, the position information extraction portion 112 performscalculation processing of the coordinates (x, y) of the upper left endpoint and the lower right end point of the image region of an imageincluding a moving object in a plurality of continuous image framesobtained in the past and stored in a buffer (not shown) provided to theimage processing device 3. Herein, the upper left end point and thelower right end point can be set so as to include the moving objectwithin the image region completely. The position information extractionportion 112 then calculates moving velocities of the respective movingobjects within the image frame using the calculated coordinates (x, y)of the upper left end points and the lower right end points in the imageframes in the past.

In Step S5, the position information extraction portion 112 calculatespredicted values of the coordinates (x, y) of the upper left end pointand the lower right end point of an image region encompassing eachtracking subject in the following image frame supplied from the datareception portion 111 on the basis of the moving velocities of therespective moving objects calculated in Step S4. An image region inwhich each tracking subject is encompassed is thus specified.

For instance, referring to FIG. 5, the position information extractionportion 112 calculates the upper left end point A₁ (X₁₁, Y₁₁) and thelower right end point B₁ (X₁₂, Y₁₂) of an image region 41 encompassingthe moving object 31 using the display screen 21 as the plane ofcoordinates. Also, the position information extraction portion 112calculates the upper left end point A₂ (X₂₁, Y₂₁) and the lower rightend point B₂ (X₂₂, Y₂₂) of an image region 42 encompassing the movingobject 32 and the upper left end point A₃ (X₃₁, Y₃₁) and the lower rightend point B₃ (X₃₂, Y₃₂) of an image region 43 encompassing the movingobject 33 in the display screen 21. The position information extractionportion 112 then supplies the cutout image generation portion 114 withthe calculated coordinates as the position information of the movingobjects. As with the example shown in FIG. 5, the image frame specifyingthe image region is set so as to be in contact with the moving object inthe longitudinal direction and in the lateral direction of thecoordinate screen.

In Step S6, the cutout image generation portion 114 calculates relativedistances in the lateral direction and the longitudinal directionbetween the respective tracking subjects using the coordinatescalculated by the position information extraction portion 112. Herein,let the display screen be the x-y coordinate plane, then the cutoutimage generation portion 114 calculates the relative distances betweenthe respective moving objects as the tracking subjects according to thedistances in the longitudinal direction and the lateral directionbetween the end points at the particular position (upper left endpoints) of the image regions specifying the moving objects and the sizesof the respective moving objects. It should be appreciated that thecalculation method of the relative distances between the respectivemoving objects is not limited to the method described above and variousother methods are also available. For example, the cutout imagegeneration portion 114 calculates the distance between the coordinatesfrom the x coordinate of the upper left end point of an image regionpresent on the left within the x-y coordinate plane to the x coordinateof the lower right end point of an image region present on the right asthe relative distance in the lateral direction. Likewise, the cutoutimage generation portion 114 calculates the distance between thecoordinates from the y coordinate of the upper left end point of animage region present on the upper side within the x-y coordinate planeand the y coordinate of the lower right end point of an image regionpresent on the lower side as the relative distance in the longitudinaldirection.

In the example of FIG. 5, the relative distance between the movingobject 31 and the moving object 33 can be expressed as a distancebetween X₃₁ and X₁₂ as the relative distance in the lateral directionand as a distance between Y₁₁ and Y₃₂ as the relative distance in thelongitudinal direction. It should be appreciated that the relativedistances are not limited to those calculated in the manner describedabove. For example, it may be configured in such a manner that thecenter points of the respective moving objects are specified and therelative distances are expressed by a distance between the center pointsin the lateral direction and a distance between the center points in thelongitudinal direction.

In Step S7, the cutout image generation portion 114 determines whetherthe relative distances at least in one of the lateral direction and thelongitudinal direction between the respective moving objects calculatedin Step S6 are larger than a predetermined threshold value(predetermined value). To be more concrete, the cutout image generationportion 114 compares the relative distances in the lateral directionbetween the respective tracking subjects with the predeterminedthreshold value and compares the relative distances in the longitudinaldirection between the respective tracking subjects with thepredetermined threshold value. The cutout image generation portion 114then determines whether the relative distances at least in one of thelateral direction and the longitudinal direction are larger than thepredetermined threshold value. Herein, the predetermined threshold valuecan be, for example, a sum value of the distances in the lateraldirection of the image regions encompassing the respective trackingsubjects in the plane of coordinates or a sum value of the distances inthe longitudinal direction of the image regions encompassing therespective tracking subjects. In this case, the cutout image generationportion 114 compares the relative distances in the lateral directionbetween the respective tracking subjects with a sum value of thedistances in the lateral direction of the image regions encompassing therespective tracking subjects and also compares the relative distances inthe longitudinal direction of the respective tracking subjects with asum value of the distances in the longitudinal direction of the imageregions encompassing the respective tracking subjects. The cutout imagegeneration portion 114 then determines whether the relative distances atleast in one of the lateral direction and the longitudinal direction arelarger than the corresponding sum value.

In a case where the cutout image generation portion 114 determines inStep S7 that the relative distances at least in one of the lateraldirection and the longitudinal direction between the respective trackingsubjects are larger than the predetermined threshold value, the flowproceeds to Step S8. In Step S8, the cutout image generation portion 114specifies cutout ranges of the respective tracking subjects and suppliesthe entire image generation portion 113 with the information of thecutout ranges as the angle-of-view information. The entire imagegeneration portion 113 then supplies the cutout image generation portion114 with images of the cutout ranges cut out from the entire imageaccording to the angle-of-view information supplied from the cutoutimage generation portion 114. The cutout image generation portion 114generates cutout images to be displayed on the cutout image displayportion 116 by applying predetermined processing to the images of thecutout ranges supplied from the entire image generation portion 113.Herein, the cutout images are enlarged by an arbitrary enlargementratio, for example, in response to a setting operation by the user. Thecutout image generation portion 114 then supplies the cutout imagedisplay portion 116 with the generated cutout images. The cutout imagedisplay portion 116 displays the cutout images supplied from the cutoutimage generation portion 114 in different cutout display windows.

In a case where the cutout image generation portion 114 determines inStep S7 that the distances among a plurality of the moving objects atleast in one of the lateral direction and the longitudinal direction areequal to or smaller than the predetermined threshold value, the flowproceeds to Step S9. In Step S9, the cutout image generation portion 114generates one cutout image including all of a plurality of the movingobjects from the entire image and supplies the cutout image displayportion 116 with the generated cutout image. In Step S9, the cutoutimage generation portion 114 specifies a cutout range including aplurality of the moving objects and supplies the entire image generationportion 113 with the information of this cutout range as theangle-of-view information. The entire image generation portion 113supplies the cutout image generation portion 114 with an image of thecutout range cut out from the entire image according to theangle-of-view information supplied from the cutout image generationportion 114. The cutout image generation portion 114 generates a cutoutimage to be displayed on the cutout image display portion 116 byapplying predetermined signal processing to the image of the cutoutrange supplied from the entire image generation portion 113. Herein, thecutout image is enlarged by an arbitrary enlargement ratio, for example,in response to a setting operation by the user. The cutout image displayportion 116 displays the cutout image supplied from the cutout imagegeneration portion 114 in one cutout display window.

The cutout image generation portion 114 is capable of setting the angleof view at the center position between the angle of view of a movingobject at the position in the closest proximity to the left end of theentire image and the angle of view of a moving object at the position inthe closest proximity to the right end of the entire image as the centerposition of the cutout image. In this case, for example, given that animage range made up of a×b (a and b are arbitrary natural numbers) dotsis the cutout range, then a distance between two points from the centerposition P2 (Xp2, Yp2) of a moving object at the right end and thecenter position P1 (Xp1, Yp1) of a moving object at the left end has tobe shorter than (a, b).

It should be appreciated, however, that the center point of the cutoutrange is not necessarily the center between P1 and P2 in the cutoutrange. In order to make motions of the moving objects in the cutoutimage smoother, it is preferable to measure an amount of movement and amoving direction (for example, in every 10 frames) and to determine anamount of movement until the next 10 frames step-by-step (for example,in a case where the cutout range has moved by 15 dots or more, thecutout range moves by 20 dots in every frame and moves by 7 dots inevery frame otherwise) rather than calculating the center position ofthe cutout range in every frame and moving the cutout range.

Accordingly, the image region of the cutout image including P1 and P2moves at a step-wise velocity in the display angle of view made up ofa×b dots.

Also, the moving direction of the center portion of the image region ofthe cutout image is determined depending on whether a plurality ofmoving objects move in a positive direction or in a negative direction.In a case where a plurality of the moving objects move in the samedirection (positive direction), the center position is moved in the samedirection. In a case where a plurality of the moving objects move in adifferent direction (negative direction), the center position is movedin a direction in which the absolute value of an amount of movement isthe largest.

In the image processing system 1, in a case where a plurality of movingobjects do not move in the same direction and the respective movingobjects eventually go out of the display angle of view, a new displayregion is increased as has been described above. In this case, a newdisplay region of a size of c×d (c and d are arbitrary natural numbers)dots is further prepared.

For example, in a case where the display region of a size of a×b dotsfor a cutout image including P1 and P2 moves from left to right, thedistance between P1 and P2 exceeds (a, b) when P2 starts to moverightward. In this case, the center position of the tracking display isset at P1 and a display region of a size of c×d dots using P2 as thecenter position of the tracking display is newly prepared. Accordingly,P1 is kept displayed in the display region of a size of a×b dots whileP2 is tracked and displayed in the display region of a size of c×d dots.

According to the image processing system 1, as has been described above,even when a moving object has moved and is no longer tracked anddisplayed in a predetermined image region, it becomes possible tocontinuously track and display this moving object by newly providing adisplay region.

Also, according to the image processing system 1, the display positionsof moving objects within the display image region are set by setting theleft end and the right end on the angle of view of an image region to bedisplayed for every image frame. In this instance, for a plurality ofmoving objects, the order of the moving objects from the left or theright can be determined on the basis of a difference of the number ofpixels between adjacent moving objects. According to the imageprocessing system 1, for example, moving objects included in the displayregion of a size of a×b dots are displayed from the right and movingobjects that do not fit inside this display region are displayed in anewly prepared display region of a size of c×d dots and the position ofthe tracking center is calculated for each display angle of view.

An example to track and display a plurality of moving objects will nowbe described. In FIG. 3 described above, moving objects 31 ₁, 32 ₁, and33 ₁ as the tracking subjects that are tracked and displayed in theentire image display window 22 on the display screen 21 travel in thesame direction indicated by an arrow T₁ at substantially the samevelocity. By repetitively performing the arithmetic processing describedabove, the cutout image generation portion 114 determines that therelative distances between the respective moving objects 31 ₁, 32 ₁, and33 ₁ are equal to or smaller than the predetermined threshold value. Thecutout image generation portion 114 then specifies a cutout range 23 ₁to display the moving objects 31 ₁, 32 ₁, and 33 ₁ within one cutoutimage. Subsequently, the cutout image generation portion 114 generates acutout image of this specified cutout range 23 ₁ and supplies the cutoutimage display portion 116 with the generated cutout image. The cutoutimage display portion 116 thus displays this cutout image in the cutoutimage display window 24. In a case where the relative distances betweenthe respective moving objects 31 ₁, 32 ₁, and 33 ₁ as the trackingsubjects are equal to or smaller than the predetermined threshold value,the user is able to recognize the moving objects 31 ₁, 32 ₁, and 33 ₁ asa group of moving objects.

Alternatively, a plurality of moving objects may be cut out anddisplayed, for example, as are shown in FIG. 6 through FIG. 9. Forinstance, the moving objects 31 ₂, 32 ₂, and 33 ₂ as the trackingsubjects to be tracked and displayed in the entire image display window22 on the display screen 21 shown in FIG. 6 travel in the same directionindicated by an arrow T₁. Herein, the travel velocities of the movingobject 31 ₂ and the moving object 32 ₂ are substantially the same.Accordingly, the cutout image generation portion 114 specifies an imageregion encompassing the moving object 31 ₂ by the upper left end pointA₄ (X₄₁, Y₄₁) and the lower right end point B₄ (X₄₂, Y₄₂). Also, themoving object 32 ₂ is specified by the upper left end point A₅ (X₅₁,Y₅₁) and the lower right end point B₅ (X₅₂, Y₅₂) and the moving object33 ₂ is specified by the upper left end point A₆ (X₆₁, Y₆₁) and thelower right end point B₆ (X₆₂, Y₆₂). The cutout image generation portion114 determines that the relative distance between the moving objects 31₂ and 32 ₂ is equal to or smaller than the predetermined thresholdvalue. In order to display the moving object 31 ₂ and the moving object32 ₂ within one cutout image, the cutout image generation portion 114sets a cutout range 23 ₂ specified by the upper left end point P₁ (X₁₀₁,Y₁₀₁) and the lower right end point Q₁ (X₁₀₂, Y₁₀₂). The cutout imagegeneration portion 114 then generates a cutout image of the set cutoutrange 23 ₂ and supplies the cutout image display portion 116 with thegenerated cutout image. The cutout image display portion 116 thusdisplays the cutout image of this cutout range 23 ₂ in a cutout imagedisplay window 24 ₂. Herein, the travel velocity of the moving object 31₂ and the moving object 32 ₂ and travel velocity of the moving object 33₂ are different. The cutout image generation portion 114 determines thatthe relative distance between at least one of the moving object 31 ₂ andthe moving object 32 ₂ and the moving object 33 ₂ is larger than thethreshold value. The cutout image generation portion 114 then sets acutout range 23 ₃ specified by the upper left end point P₂ (X₂₀₁, Y₂₀₁)and the lower right end point Q₂ (X₂₀₂, Y₂₀₂).

Subsequently, the cutout image generation portion 114 generates a cutoutimage of the cutout range 23 ₃ including the moving object 33 ₂ andsupplies the cutout image display portion 116 with the generated cutoutimage. The cutout image display portion 116 displays the cutout image ofthis cutout range 23 ₃ in the cutout image display window 24 ₁.

FIG. 7 through FIG. 9 show examples of a case where the moving object 31₂ and the moving object 32 ₂ traveling in the same direction indicatedby an arrow T₁ and the moving object 33 ₂ traveling in a direction (adirection indicated by an arrow T₂) opposite to the travel direction ofthe moving object 31 ₂ and the moving object 32 ₂ are displayed in theentire image display window 22 on the display screen 21.

In the example shown in FIG. 7, the cutout image generation portion 114determines that the relative distance between the moving object 31 ₂ andthe moving object 32 ₂ is equal to or smaller than the predeterminedthreshold value and sets a cutout range 23 ₂ specified by the upper leftend point P₁ (X₁₀₁, Y₁₀₁) and the lower right end point Q₁ (X₁₀₂, Y₁₀₂)in order to display the moving object 31 ₂ and the moving object 32 ₂within one cutout image. The cutout image generation portion 114 thengenerates a cutout image of the set cutout range 23 ₂ and supplies thecutout image display portion 116 with the generated cutout image. Thecutout image display portion 116 thus displays the cutout image of thecutout range 23 ₂ in the cutout image display window 24 ₂. Also, thecutout image generation portion 114 determines that the relativedistance between at least one of the moving object 31 ₂ and the movingobject 32 ₂ and the moving object 33 ₂ is larger than the predeterminedthreshold value. The cutout image generation portion 114 then sets acutout range 23 ₃ specified by the upper left end point P₂ (X₂₀₁, Y₂₀₁)and the lower right end point Q₂ (X₂₀₂, Y₂₀₂) in order to display thecutout image including the moving object 33 ₂ in a cutout image displaywindow different from the cutout image display window 24 ₂. The cutoutimage generation portion 114 then generates a cutout image of thespecified cutout range 23 ₃ and supplies the cutout image displayportion 116 with the generated cutout image. The cutout image displayportion 116 thus displays the cutout image of the cutout range 23 ₃ inthe cutout image display window 24 ₁.

Also, as is shown in FIG. 8, when the moving object 31 ₂ and the movingobject 32 ₂ traveling in the direction indicated by an arrow T1 and themoving object 33 ₂ traveling in the direction indicated by an arrow T₂come closer to each other, the cutout image generation portion 114determines that the relative distance between at least one of the movingobject 31 ₂ and the moving object 32 ₂ and the moving object 33 ₂ isequal to or smaller than the predetermined threshold value. The cutoutimage display portion 116 then displays the moving object 31 ₂, themoving object 32 ₂, and the moving object 33 ₂ in a cutout image displaywindow 24 ₃ having an enlarged display region.

Assume that the moving object 31 ₂ and the moving object 32 ₂ travelingin the direction indicated by the arrow T₁ and the moving object 33 ₂traveling in the direction indicated by the arrow T₂ come closer to eachother as is shown in FIG. 8 and then move apart later as is shown inFIG. 9. In this instance, the cutout image generation portion 114determines that the relative distance between at least one of the movingobject 31 ₂ and the moving object 32 ₂ and the moving object 33 ₂ islarger than the predetermined threshold value. Accordingly, the cutoutimage display portion 116 displays the moving object 31 ₂ and the movingobject 32 ₂ in the cutout image display window 24 ₁ and also displaysthe moving object 33 ₂ in the cutout image display window 24 ₂.

An example of the moving object tracking display by the image processingsystem 1 will now be described. FIG. 10 shows a display screen 51 atSXGA of 1280×1024 dots. The display screen 51 has an entire imagedisplay region 501 as a entire image display window, a camera controlpanel 502, and a first detailed image display region 503, a seconddetailed image display region 504, and a maximum display region 505 ascutout image display windows. The display screen 51 also has anoperation button 506 including a REW (rewind) button, a REC (record)button, a PLAY (play) button, a STOP (stop) button, and an FF (fastforwarding) button for each image being displayed, and a display portion507 on which a file name and a time code are displayed.

The entire image display region 501 is a display region made up of640×360 dots and it is a region in which tracking subjects aredisplayed. Both of the first detailed image display region 503 and thesecond detailed image region display region 504 are regions made up of600×450 dots. The maximum display region 505 is a region made up of1250×480 dots.

In a case where a plurality of tracking subjects are displayed in theentire image display region 501, a plurality of the tracking subjectsdetermined as having the relative distances between the respectivetracking subjects larger than the predetermined threshold value aredivided, so that a plurality of the tracking subjects are enlarged anddisplayed in the first detailed image display region 503 and the seconddetailed image display region 504.

Also, in a case where a plurality of tracking subjects are displayed inthe entire image display region 501, a plurality of the trackingsubjects determined as having the relative distances between therespective tracking subjects equal to or smaller than the predeterminedthreshold value are displayed in the maximum display region 505.

The camera control panel 502 has operations buttons to control thecamera portion 10, for example, buttons for zoom, focus, iris(diaphragm), shutter, gain, brightness (luminance), white balance, pan,and tilt control.

FIG. 11 through FIG. 13 are views showing examples of a display screenshowing moving objects 34, 35, and 36 present about 3 km ahead theposition of the camera portion 10 and captured by the camera portion 10formed of an HD camera having output pixels at 1920×1080 dots and set atthe angle of view of 10 degrees.

FIG. 11 is a view showing a display screen 51 showing a region having ahorizontal width of 520 m and including the moving objects 34 through 36as tracking subjects having horizontal widths of 50 m, 10 m, and 30 m,respectively, in the entire image display region 501. In the exampleshown in FIG. 11, the horizontal width of 340 m is given as thepredetermined threshold value. The moving objects 34 through 36 areincluded completely within an image region having the horizontal widthof 340 m. More specifically, in the entire image display region 501, adistance between X₆₀₁, which is the x coordinate of the upper left endpoint P₆ specifying the moving object 36 at the leftmost position, andX₄₀₂, which is the x coordinate of the lower right end point Q₄specifying the moving object 34 at the rightmost position, represents adistance 340 m or less.

The cutout image generation portion 114 generates a cutout image(detailed image) of the same pixel size, which is an image region havingthe horizontal width of 340 m to completely include the moving objects34 through 36 and cut out from the entire image display region 501, andsupplies the cutout image display portion 116 with this cutout image.The cutout image display portion 116 thus displays the cutout imagesupplied from the cutout image generation portion 114 in the maximumdisplay region 505.

There is a case where none of the moving objects are completely includedin the image region having the horizontal width of 340 m in the entireimage display region 501. In an example shown in FIG. 12, none of themoving objects 34 through 36 are completely included in the image regionhaving the horizontal width of 340 m when cut out from the entire imagedisplay region 501. More specifically, in the entire image displayregion 501, a distance between X₆₀₁ which is the x coordinate of theupper left end point P₆ specifying the moving object 36 at the leftmostposition, and X₄₀₂, which is the x coordinate of the lower right endpoint Q₄ specifying the moving object 34 at the rightmost position,represents a distance exceeding 340 m.

The cutout image generation portion 114 specifies a cutout rangeincluding the moving object 36 from the entire image display region 501and generates a cutout image for the moving object 36 to be displayed inthe first detailed image display region 503. Also, the cutout imagegeneration portion 114 specifies a cutout range including the movingobjects 34 and 35 and generates a cutout image for the moving objects 34and 35 to be displayed in the second detailed image display region 504.The cutout image display portion 116 then displays the cutout imageincluding the moving object 36 in the first detailed image displayregion 503 and also displays the cutout image including the movingobjects 34 and 35 in the second detailed image display region 504.

Also, in an example shown in FIG. 13, all of the moving objects 34through 36 are completely included within a region having a horizontalwidth of 520 m. Herein, the user is able to set the digital ×2 zoom inthe imaging mode of the camera portion 10 formed of an HD camera whendetailed images are displayed. Herein, in a case where a distancebetween X₆₀₁ and X₄₀₂ in the entire image display region 501 exceeds ahorizontal width of 260 m, the cutout image display region 116 displaysthe moving object 36 in the first detailed image display region 503 andalso displays the moving objects 34 and 35 in the second detailed imagedisplay region 504.

FIG. 14 is a flowchart used to describe a moving object tracking anddisplaying processing operation according to latitude and longitudeinformation by the image processing system 1. This moving objecttracking and displaying processing according to the latitude andlongitude information is suitable to a case where a captured image is astill image.

In Step S11, the position information extraction portion 112 starts themoving object tracking and displaying processing according to thelatitude and longitude information.

In Step S12, the position information extraction portion 112 calculatesthe positions of the image transmission device 2 and a reference subjectX according to the latitude and the longitude in the image framesupplied from the data reception portion 111, for example, as is shownin FIG. 15. The installment position of the image transmission device 2is expressed by the center position R₀ (Lon0, Lat0) of the imagetransmission device 2. The center position R_(x) of the referencesubject X is expressed by R_(x) (LonX, LatX). Also, in Step S12, theposition information extraction portion 112 calculates the azimuth angleC (P, Q) using P [dg] and Q [deg] in such a manner that the imagetransmission device 2 falls on the origin of the line of sight to thecenter position R_(x) (LonX, LatX) of the reference subject X.

In Step S13, as is shown in FIG. 16, the position information extractionportion 112 sets the pixel value and an origin C₀ as an initial value soas to fall on the center of the horizontal and vertical angles of viewsin an entire image display window 61 displayed by the image processingdevice 3. The origin C₀ (0, 0) is calculated in accordance with C (P,Q)=C (m/2, n/2).

In Step S14, the position information extraction portion 112 calculatesan angle (θ, φ) produced between the line of sight of the imagetransmission device 2 to the center position R_(x) of the referencesubject X and a direction from the image transmission device 2 to thecenter position R₁ of a moving object 31D. The position informationextraction portion 112 calculates a distance from the image transmissiondevice 2 to the center position R₁ of the moving object 31D using thecenter position R₀ (Lon0, Lat0) of the image transmission device 2 andthe center position R₁ (Lon1, Lat1) of the moving object 31D.

In Step S15, the position information extraction portion 112 specifiesan image region 41D of an image including the moving object 31D on thebasis of the angle (θ, φ) produced between the line of sight of theimage transmission device 2 and the direction from the imagetransmission device 2 to the center position R₁ of the moving object 31Dand the distance from the image transmission device 2 to the centerposition R₁ of the moving object 31D calculated in Step S14. The imageregion 41D specified by the position information extraction portion 112is not limited to a case where it is specified so that the region frameis in contact with the moving object 31D as is shown in FIG. 15. It maybe configured in such a manner that the position information extractionportion 112 specifies the image range 41D as a small region so that themoving object 31D will be displayed large, for example, in a case wherea distance from the image transmission device 2 to the moving object 31Dis long. Alternatively, it may be configured in such a manner that theposition information extraction portion 112 specifies the image region41D as a large region so that the moving object 31D is displayed small,for example, in a case where a distance from the image transmissiondevice 2 to the moving object 31D is short.

In Step S16, the position information extraction portion 112 performsprocessing to covert the angular coordinate to the pixel coordinate inorder to display the image of the image region 41D calculated in StepS15 in the cutout image display window.

In Step S17, the position information extraction portion 112 appliescalculation processing in the same manner as in Step S12 through StepS16 to a plurality of moving objects and specifies image regions ofmoving objects as tracking subjects.

In Step S18, the position information extraction portion 112 performscalculation processing of the coordinates (x, y) of the upper left endpoint and the lower right end point in an image region of an imageincluding the moving objects in a plurality of continuous image framesobtained in the past and stored in a buffer (not shown) provided to theimage processing device 3. The position information extraction portion112 then calculates moving velocities of the respective moving objectsin the image frame using the calculated coordinates (x, y) of the upperleft end point and the lower right end point in the image frames in thepast. Subsequently, the position information extraction portion 112calculates the predicted values of the coordinates (x, y) of the upperleft end point and the lower right end point of the image regionincluding the moving objects in the following image frame supplied fromthe date reception portion 111. Accordingly, the position informationextraction portion 112 specifies the coordinates of the upper left endpoint and the lower right end point of the image region including aplurality of anticipated moving objects.

In Step S19, the cutout image generation portion 114 calculates relativedistances in the longitudinal direction and the lateral directionbetween the respective moving objects among a plurality of the movingobjects on the basis of the coordinate values specified by the positioninformation extraction portion 112.

In Step S20, the cutout image generation portion 114 determines whetherthe relative distances at least in one of the lateral direction and thelongitudinal direction between the respective moving objects calculatedin Step S19 are larger than a predetermined threshold value. In a casewhere the cutout image generation portion 114 determines in Step S20that the relative distances at least in one of the lateral direction andthe longitudinal direction are larger than the predetermined thresholdvalue, the flow proceeds to Step S21. In Step S21, the cutout imagegeneration portion 114 specifies cutout ranges to display the respectivemoving objects in different cutout image display windows. The cutoutimage generation portion 114 then generates cutout images of thespecified cutout ranges and supplies the cutout image display portion116 with the generated cutout images. The cutout image display portion116 thus displays the cutout images of the specified cutout ranges indifferent cutout image display windows. Also, in a case where the cutoutimage generation portion 114 determines in Step S20 that the relativedistances at least in one of the lateral direction and the longitudinaldirection between the respective moving objects are equal to or smallerthan the predetermined threshold value, the flow proceeds to Step S22.In Step S22, the cutout image display portion 116 displays the cutoutimage including a plurality of the moving objects in one cutout imagedisplay window according to the information of the cutout range suppliedfrom the cutout image generation portion 114.

According to the image processing system 1 performing the processing asabove, by displaying a plurality of tracking subjects in the entireimage display window and the cutout image display window(s), it becomespossible to obtain detailed images of the tracking subjects that havenot been obtained by the wide angle imaging while keeping track of thepositional relation among the tracking subjects.

Also, according to the image processing system 1, even in a case where aplurality of tracking subjects are not displayed in one cutout imagedisplay window, it becomes possible to display a plurality of thetracking subjects in a plurality of cutout image display windows byspecifying the cutout ranges of the respective tracking subjects.

Also, according to the image processing system 1, in a case where therelative distances among a plurality of the tracking subjects are equalto or smaller than the predetermined threshold value, it becomespossible to track a plurality of the tracking subjects as a group bydisplaying all the tracking subjects in one cutout image display window.Accordingly, even when there are a large number of tracking subjects, itbecomes possible to keep track of the tracking subjects preciselyaccording to the positional relation among the tracking subjects beingdisplayed in the entire image display window.

Also, according to the image processing system 1, for example, even in acase where a plurality of tracking subjects cross with each other and itbecomes difficult to image a given moving object hidden behind anothermoving object, it becomes possible to cut out and display this hiddentracking subject according to the positions of the moving object inimage frames in the past.

An image processing system according to another embodiment to which thepresent invention is applied will now be described. In the imageprocessing system according to another embodiment of the presentinvention, a single image transmission device on which the swivel baseis mounted generates a panoramic entire image by joining images obtainedby swiveling in the lateral direction and in the longitudinal directionby one screen. Alternatively, it generates a panoramic entire image byimaging continuous regions using a plurality of image transmissiondevices and joining the generated images.

In the image processing system according to another embodiment of thepresent invention, for example, as is shown in FIG. 17, of a pluralityof generated images (images 701 ₁, 701 ₂, 701 ₃, . . . , 701 _(n), . . ., 70 n ₁, 70 n ₂, 70 n ₃, . . . , 70 n _(n-1), and 70 n _(n)), apredetermined number of continuous images (for example, images 701 ₁,701 ₂, 701 ₃, 701 ₄, 701 ₅, and 701 ₆) are displayed in an entire imagedisplay window 702. Also, images of predetermined cutout ranges in theimages being displayed in the entire image display window 702 aredisplayed in the cutout image display window 703.

In the image processing system according to another embodiment of thepresent invention, tracking subjects are tracked and displayed in thepanoramic entire image as above. With the image processing systemaccording to another embodiment of the present invention, it becomespossible to track and display a moving object that disappears from animage captured by a single camera portion.

FIG. 18 is a view showing the configuration of an image processingsystem 200 according to another embodiment to which the presentinvention is applied. In the image processing system 200, componentssame as those of the image processing system 1 are labeled with the samereference numerals and detailed descriptions are omitted herein. As isshown in FIG. 18, the image processing system 200 includes an imagetransmission device 2A and an image processing device 3A connected toeach other via a network 4.

FIG. 19 is a view showing the internal configuration of the imagetransmission device 2A and the image processing device 3A. As is shownin FIG. 19, in addition to the configuration of the image transmissiondevice 2, the image transmission device 2A further includes a swivelbase 15 and a control portion 14 that controls the swivel base 15.

The swivel base 15 has the camera portion 10 mounted thereon and swivelsin the lateral direction and the longitudinal direction by one screenunder the control of the control portion 14. The control portion 14supplies the data transmission portion 13 with position information ofthe swivel base 15. The camera portion 10 captures continuous images perscreen according to a swivel operation of the swivel base 15 controlledby the control portion 14. In this case, the control portion 14 controlsa moving distance per pulse of the camera portion 10 in association withthe movement of the swivel base 15 according to the angle information.

The pulse value on the basis of the swivel angle of view of the cameraportion 10 and the moving distance of the swivel base 15 is set in sucha manner that the center of the swivel angle of view of the cameraportion 10 falls on the origin (0, 0).

The imaging signal processing portion 102 appends position informationof the swivel base 15 when the center position of an image is capturedto image data of the image captured by the imaging portion 101. Theimaging signal processing portion 102 supplies the encoder 103 and themoving object and stationary object detection portion 104 with the imagedata appended with the position information of the swivel base 15. Theimage transmission device 2A then transmits the image data appended withthe position information of the swivel base 15 to the image processingdevice 3A.

Although it is not shown in the drawing, the image processing device 3Aincludes a CPU, a working RAM, a ROM that stores an image processingprogram according to one embodiment of the present invention, data, andso forth. By running the image processing program on the CPU, a seriesof the processing is performed by the image processing device 3A.

A panoramic image generation portion 113A provided to the imageprocessing device 3A performs decoding processing of the image datareceived from the image transmission device 2A via the data receptionportion 111 and generates a panoramic entire image using the decodedimage data. Subsequently, the panoramic image generation portion 113Asupplies a panoramic image display portion 115A and a cutout imagegeneration portion 114A with the generated panoramic entire image. Thepanoramic image generation portion 113A performs processing to joinimage data received from the image transmission device 2A and generatesa panoramic entire image by reducing the joined images later.Alternatively, it may be configured in such a manner that the panoramicimage generation portion 113A performs processing to reduce a pluralityof items of image data received from the image transmission device 2Afirst and then generates a panoramic entire image by joining a pluralityof items of reduced image data. The panoramic image generation portion113A then supplies the cutout image generation portion 114A and thepanoramic image display portion 115A with the generated panoramic entireimage.

The position information extraction portion 112 extracts positioninformation of moving objects in the image by performing predeterminedarithmetic processing on the basis of the image data supplied from thedata reception portion 111 and metadata as detection information of themoving objects and the stationary objects. It may be configured in sucha manner that the position information extraction portion 112 takes inthe position information of the moving objects from the outside, such asthe GPS, and converts the position information of the moving objects tothe position information in the image.

The position information extraction portion 112 is capable ofcalculating a distance between two points, for example, in accordancewith the Hubeny's distance formula expressed by Mathematical Expression(1) below from two points whose latitude and longitude are specified bythe GPS:

D=sqrt((M*dP)*(M*dP)+(N*cos(P)*dR)*(N*cos(P)*dR))   (1)

where D is a distance between two points (m), P is a mean latitude oftwo points (radian), dP is a difference in latitude between two points,dR is a difference in longitude between two points, M is a meridianradius of curvature, and N is a prime vertical radius of curvature.

Also, a calculation method of a distance between two points specified bythe GPS may be referred to “TOTAL INVERSE SOLUTIONS FOR THE GEODESIC ANDGREAT ELLIPTIC”, B. R. Bowring, Survey Review, 33, 261 (July 1996)461-476.

The cutout image generation portion 114A specifies cutout rangesincluding moving objects as tracking subjects and generates cutoutimages of the cutout ranges. The cutout image generation portion 114Athen supplies the cutout image display portion 116A with the cutoutimages. Herein, the cutout image generation portion 114A performs theprocessing same as the processing performed by the cutout imagegeneration portion 114 described above. More specifically, in a casewhere the relative distances are larger than a predetermined thresholdvalue, the cutout image generation portion 114A specifies cutout rangesto display the respective moving objects in different cutout imagedisplay windows and generates cutout images of respective images of thecutout ranges. Subsequently, the cutout image generation portion 114Asupplies the cutout image display portion 116A with the cutout images ofthe respective specified cutout ranges. Also, in a case where therelative distances are equal to or smaller than the predeterminedthreshold value, the cutout image generation portion 114A specifies acutout range to display the moving objects on one cutout display screenas a group of moving objects. The cutout image generation portion 114Athen generates a cutout image of the specified cutout range and suppliesthe cutout image display portion 116A with the generated cutout image.

The cutout image display portion 116A thus displays the cutout imagesupplied from the cutout image generation portion 114A in the cutoutimage display window. As with the cutout image display portion 116, thecut out image display portion 116A displays respective moving objects indifferent cutout image display windows in a case where the relativedistances are larger than the predetermined threshold value.

For example, as is shown in FIG. 20, the panoramic image generationportion 113A generates a panoramic entire image 704 specified by theangle coordinates having the center at (0, 0) and extending by −α degree(leftward) and +α degree (rightward) on the abscissa and by −β degree(upward) and +β degree (downward) on the ordinate. In this manner, thepanoramic image generation portion 113A controls a moving distance perpulse using an angle. The cutout image generation portion 114A specifiesa cutout range 705 including a moving object 32D as a tracking subject.Herein, the cutout image generation portion 114A specifies the cutoutrange 705 in such a manner that the moving object 32D comes in closestproximity to the center of the cutout range 705. The cutout imagegeneration portion 114A generates a cutout image of the cutout range 705and supplies the cutout image display portion 116 with the generatedcutout image.

It may be configured in such a manner that the image processing system200 includes, for example, an image transmission device 2B having n (nis an integer equal to or larger than 2) image transmission portions2B₁, 2B₂, . . . , and 2B_(n) as shown in FIG. 21 instead of the imagetransmission device 2A. In this case, each of the image transmissionportions 2B₁, 2B₂, . . . , and 2B_(n) generates images of continuousregions and transmits a plurality of the images to the image processingdevice 3A via the data control portion 17. The image processing device3A generates a panoramic entire image from a plurality of the continuousimages received from the data control portion 17.

Regarding the image transmission portions 2B₁, 2B₂, . . . , and 2B_(n)shown in FIG. 21, components same as those of the image transmissiondevice 2A shown in FIG. 19 are labeled with the same reference numeralsand detail descriptions are omitted herein. Also, the image transmissionportion 2B₁ alone will be described in the following on the assumptionthat the image transmission portions 2B₁, 2B₂, . . . , and 2B_(n) are ofthe same configuration.

The image transmission portion 2B₁ includes a camera portion 10, animage data processing portion 11B, an external sensor 12, and a bufferportion 16. The image data processing portion 11B includes an encoder103 ₁ as a first encoder, an encoder 103 ₂ as a second encoder, a movingobject and stationary object detection portion 104, a metadatageneration portion 105, and a size changing portion 106.

An image frame generated by the imaging signal processing portion 102 iscorrelated with the number assigned to the camera portion 10 (cameranumber) and a frame number.

The encoder 103 ₁ generates first compressed image data by applyingencoding, for example, by the Motion JPEG on a plurality of image framesat first resolution outputted from the camera portion 10.

The size changing portion 106 generates image frames at secondresolution (for example, QVGA) lower than the first resolution byapplying processing, such as skipping and interpolation, to the imageframes at the first resolution outputted from the camera portion 10.

The encoder 103 ₂ generates second compressed image data by applyingencoding processing to a plurality of the image frames at the secondresolution outputted from the size changing portion 106. As with theencoder 103 ₁, the encoder 103 ₂ also performs compression coding, forexample, by the Motion JPEG.

The image data at the first resolution from the encoder 103 ₁, the imagedata at the second resolution from the encoder 103 ₂, and the metadatafrom the metadata generation portion 105 are supplied to the bufferportion 16. The buffer portion 16 temporarily stores the supplied datain a buffer memory and supplies the data control portion 17 with thedata later.

The data control portion 17 puts n items of image data at the firstresolution, n items of image data at the second resolution, and metadatasupplied from the respective image transmission portions 2B₁, 2B₂, . . ., and 2B_(n) into a file and transmits the file to the image processingdevice 3A.

The data reception portion 111 supplies the panoramic image generationportion 113A with the image data at the first resolution and the imagedata at the second resolution received from the data control portion 17in the image transmission device 2B. The data reception portion 111supplies the position information extraction portion 112 with the imagedata at the first resolution and the metadata received from the datacontrol portion 17 in the image transmission device 2B.

The panoramic image generation portion 113A performs decoding processingof the image data at the first resolution and the image data at thesecond resolution. The panoramic image generation portion 113A suppliesthe cutout image generation portion 114 with the decoded image data atthe first resolution. Also, the panoramic image generation portion 113Agenerates a panoramic entire image using the decoded image data at thesecond resolution. The panoramic image generation portion 113A thensupplies the panoramic image display portion 115A with the image data ofthe panoramic entire image. The panoramic image display portion 115Athus displays the panoramic entire image supplied from the panoramicimage generation portion 113A in the entire image display window.

The cutout image generation portion 114A generates cutout images oftracking subjects on the basis of relative distances between therespective tracking subjects in the image data at the first resolutionaccording to the position information extracted by the positioninformation extraction portion 112.

In this manner, in a case where the panoramic entire image is generatedfrom the image data generated by a plurality of the camera portions 10,given that the angle of view per used lens is known, then a value foundby subtracting overlapping regions from the number of lenses×lens angleof view is defined as the angle of view of the entire region of thepanoramic entire image. Also, the angle of view per camera portion isthe value found by dividing the angle of view of the entire region bythe number of lenses. Also, it is possible to calculate the cutout rangefrom the angle per pixel in the panoramic entire image.

It should be noted that the image processing system 200 is capable oftracking and displaying tracking subjects by obtaining the positioninformation (hereinafter, referred to as the position information Pn) ofthe tracking subjects, for example, according to the GPS, the beaconinformation, a radar, and so forth.

In this case, the image processing device 3A receives image data, forexample, appended with the position information of the swivel base 15,from the image transmission device 2A. The position information Pn isobtained out of synchronization with the image data from the outside bythe position information extraction portion 112.

The position information extraction portion 112 converts the positioninformation Pn to a radian format and supplies the cutout imagegeneration portion 114A with the position information Pn converted intothe radian format.

The cutout image generation portion 114A calculates the azimuth angle ofthe coordinate (x, y) on the latitude and longitude of the centerposition P of a moving object specified when calculating a distancebetween the origin O and the center position P of the moving object andcalculates a difference from the azimuth angle of the origin O. Thisdifference is the center of the angle of view to be cut out. The cutoutimage generation portion 114A searches for an approximate image andspecifies a cutout range to display the approximate image together witha plurality of nearby images.

The cutout image generation portion 114A calculates an angle producedwith the origin O each time the position information Pn is supplied andcalculates the coordinate of the position corresponding to the positioninformation Pn. The image processing device 3A is thus capable ofdisplaying the moving object continuously.

In this manner, the image processing system 200 calculates a distancebetween two points, that is, a distance from the origin to the movingobject on the basis of two pieces of the latitude and longitudeinformation of the origin 0 and the position information Pn.Accordingly, in a case where the sizes of the tracking subjects areknown in advance or can be predicted, the image processing system 200 iscapable of enlarging and displaying the tracking subjects by controllinga zooming operation of the camera portion 10.

The image processing system 200 is capable of displaying display screensas shown, for example, in FIG. 22 through FIG. 24. A display screen 706shown in FIG. 22 and FIG. 23 displays a video, for example, a soccergame. A panoramic entire image display window 707 scroll-displays thesoccer field as a panoramic entire image. The cutout image generationportion 114A specifies a cutout range according to motions of a linejudge C₁ and a line judge C₂ that are the tracking subjects. In anexample of FIG. 22, the cutout image generation portion 114A determinesthat a distance L₁ in the lateral direction between the line judge C₁and the line judge C₂ is equal to or smaller than the predeterminedthreshold value. The cutout image generation portion 114A then specifiesa cutout range 708 including the line judge C₁ and the line judge C₂ andgenerates a cut out image, which is an enlarged image of the cutoutrange 708. The cutout image display portion 116A thus displays thecutout image including the line judge C₁ and the line judge C₂ in acutout image display window 709.

In a display example shown in FIG. 23, the cutout image generationportion 114A determines that a distance L₂ in the lateral directionbetween the line judge C₁ and the line judge C₂ is larger than thepredetermined threshold. The cutout image generation portion 114A thenspecifies a cutout range 710 including the line judge C₁ and a cutoutrange 711 including the line judge C₂. The cutout image generationportion 114A generates a cutout image, which is an enlarged image of thecutout range 710, and a cutout image, which is an enlarged image of thecutout range 711. The cutout image display portion 116A thus displaysthe cutout image, which is an enlarged image of the cutout range 710,and the cutout image, which is an enlarged image of the cutout range711, in a cutout image display window 712 and a cutout image displaywindow 713, respectively.

In the display examples shown in FIG. 22 and FIG. 23, it is possible toset the center position between the position of the line judge C₁ andthe position of the line judge C₂ to the center position of the displayangle of view of the entire image being displayed in the panoramicentire image display window 707.

In a case where the image processing system 200 generates a panoramicentire image using images generated by a plurality of the imagetransmission portions 2B₁, 2B₂, . . . , and 2B_(n), it shouldappreciated that the method of specifying the center position of thepanoramic entire image is not limited to the method using the centerposition of two points as described above. In this case, the numbers ofmoving objects within the respective image frames generated by therespective image transmission portions are summed up. Then, the maximumnumber (mode value) of the numbers of moving objects within the imageframes is specified and the display angle of view is specified bysetting the center position to the camera portion 10 that captured asmany moving objects as the mode value. Subsequently, an amount ofmovement of the camera portion 10 by which the display angle of view ismoved gradually is calculated. The imaging direction is set to move, forexample, in a direction of the camera portion 10 having the secondlargest value next to the mode value. An amount of movement of thecamera portion 10 is, for example, 4 degrees/sec (560 pixels) at themaximum and normally about 140 pixels. An amount of movement per frameis about 10 pixels and about 40 pixels at the maximum.

In the display examples shown in FIG. 22 and FIG. 23, the panoramicimage generation portion 113A calculates the center of a virtual angleof view from the position of the line judge C₁ and the position of theline judge C₂. The panoramic image display portion 115A scroll-displaysthe panoramic entire image in the panoramic entire image display window707 according the motion of the line judges that differs from one sceneto another.

Also, a display screen 720 shown in FIG. 24 is an example of the displayscreen displayed by a personal computer. In the display screen 720, theposition information (height, angles (pan and tilt) from the cameraportion 10, latitude, longitude, and so forth) of a moving object isdisplayed in a region 724. Also, the position information (latitude,longitude, angle in the line of sight, height, and so forth) of theinstalled position of the camera portion 10 is displayed in a region725. In a case where the monitoring camera detects a moving object C₃,such as a suspicious individual, the display screen 720 displays acutout image of a cutout range 722 including the moving object C₃ in acutout image display window 723. In this manner, the image processingsystem 200 is used as a monitoring system used for monitoring bydisplaying a moving object in the cutout image display window 723. Inthe example of FIG. 24, the image can be either a moving image or astill image. In a case where the image is a still image, the imagedisplayed in the cutout image display window 723 is updated in everyarbitrary preset time.

As has been described, the image processing systems according to theembodiments of the present invention track and display a plurality ofmoving objects simultaneously by using the entire image and a cutoutimage(s). Accordingly, the user is able to keep track of the positionalrelation between the respective tracking subjects among a plurality ofthe tracking subjects in the entire image whereas the user is able tokeep track of a whole picture of the tracking subjects, which is notkept track of by the mere wide angle imaging, in detail in the cutoutimage.

Also, with the image processing systems according to the embodiments ofthe present invention, even in a case where a specific tracking subjectis distant from a group of a plurality of tracking subjects by adistance larger than the predetermined threshold, the user will not losetrack of the tracking subject by newly preparing a display region of thepredetermined angle of view.

Also, with the image processing systems according to the embodiments ofthe present invention, the user is able to track a plurality of trackingsubjects having the relative distances equal to or smaller than thepredetermined threshold value as a group. Accordingly, even when thereare a large number of tracking subjects, it is possible to keep track ofall the tracking subjects.

It should be appreciated that the present invention is not limited tothe embodiments described above and it goes without saying that variousmodifications are possible within the scope of the invention.

In the embodiments described above, the image processing device 3includes the entire image generation portion 113 and the entire imagedisplay portion 115 to control the entire image of a predetermined rangeto be displayed in the entire image display window. Also, the imageprocessing device 3 includes the cutout image generation portion 114 andthe cutout image display portion 116 to control a plurality of trackingsubjects included in the entire image to be enlarged and displayed inthe cutout image display window(s). It should be appreciated, however,that the present invention is not limited to the embodiments as above.For example, an image generation portion, an image display portion, andso forth may be provided to the outside of the image processing deviceas long as the entire image display control processing and the cutoutimage display control processing described above are performed.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2008-309969 filedin the Japan Patent Office on Dec. 4, 2008, the entire contents of whichis hereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An image processing device comprising: an entire image displaycontrol portion that performs control to display an entire image of apredetermined region in an entire image display window; and a cutoutimage display control portion that performs control to enlarge aplurality of tracking subjects included in the entire image and displaythe tracking subjects in a cutout image display window, wherein thecutout image display control portion performs the control in such amanner that one cutout image including the plurality of trackingsubjects is displayed in the cutout image display window in a case whererelative distances among the plurality of tracking subjects are equal toor smaller than a predetermined value and performs the control in such amanner that two cutout images including the respective tracking subjectsare displayed in the cutout image display window in a case where therelative distances among the plurality of tracking subjects are largerthan the predetermined value.
 2. The image processing device accordingto claim 1 further comprising: an operation portion that specifies theplurality of tracking subjects from the entire image being displayed inthe entire image display window in response to an operation by a user.3. The image processing device according to claim 1, wherein therelative distances are calculated on the basis of one or both of sizesof the tracking subjects within the entire image and distances among thetracking subjects within the entire image.
 4. The image processingdevice according to claim 2, wherein the cutout image display controlportion changes an enlargement ratio of the cutout image in response toan operation from the operation portion.
 5. An image processing systemcomprising: an image transmission device having a camera portion thatgenerates image data, an image data processing portion that generatescompressed image data by encoding the image data generated by the cameraportion, and a data transmission portion that transmits the compressedimage data generated by the image data processing portion; and an imageprocessing device having a data reception portion that receives thecompressed image data, an entire image display control portion thatperforms control to display an entire image of a predetermined rangegenerated from image data obtained by decoding the compressed image datareceived at the data reception portion in an entire image displaywindow, and a cutout image display control portion that performs controlto enlarge a plurality of tracking subjects included in the entire imageand display the tracking subjects in a cutout image display window,wherein the cutout image display control portion performs the control insuch a manner that one cutout image including the plurality of trackingsubjects is displayed in the cutout image display window in a case whererelative distances among the plurality of tracking subjects are equal toor smaller than a predetermined value and performs the control in such amanner that two cutout images including the respective tracking subjectsare displayed in the cutout image display window in a case where therelative distances among the plurality of tracking subjects are largerthan the predetermined value.
 6. The image processing system accordingto claim 5, wherein the image transmission device further includes aswivel base on which the camera portion is mounted and which performs aswivel operation; the camera portion generates a plurality of items ofimage data of continuous regions according to the swivel operation ofthe swivel base; and the entire image display control portion performsthe control in such a manner that a panoramic entire image generatedfrom the plurality of items of image data of the continuous regions isdisplayed in the entire image display window.
 7. The image processingsystem according to claim 5, wherein the image transmission device isprovided in a plural form; the camera portion in each of a plurality ofthe image transmission devices generates a plurality of items of imagedata of continuous regions; and the entire image display control portionperforms the control in such a manner that a panoramic entire imagegenerated from the plurality of items of image data of continuousregions is displayed in the entire image display window.
 8. An imageprocessing method comprising the steps of: performing control to displayan entire image of a predetermined region in an entire image displaywindow; and performing control to enlarge a plurality of trackingsubjects included in the entire image and display the tracking subjectsin a cutout image display window, wherein, in the step of displaying thetracking subjects, the control is performed in such a manner that onecutout image including the plurality of tracking subjects is displayedin the cutout image display window in a case where relative distancesamong the plurality of tracking subjects are equal to or smaller than apredetermined value and the control is performed in such a manner thattwo cutout images including the respective tracking subjects aredisplayed in the cutout image display window in a case where therelative distances among the plurality of tracking subjects are largerthan the predetermined value.
 9. (canceled)